Analyzing Inter-Job Contention in Dragonfly Networks

Interconnection networks are increasing in importance as node counts increase in high-end machines. To achieve better application performance, newer supercomputers frequently have interconnects with more connections, higher bandwidth, and lower diameter. One example of such an interconnect is a dragonfly topology, which has appeared in multiple recent supercomputers. Adaptive routing and high bandwidth on dragonfly networks leads to the belief that sharing of the network between jobs will not lead to performance degradation. In this paper, we analyze the performance of a production HPC application, MILC, on a dragonfly-based Cray supercomputer. We find that, in fact, the performance of MILC varies by a factor of more than three, and that the performance variation is due to communication delays from network interference. First, we analyze a communication trace of MILC to relate per-rank delays to network activity. Then we use machine learning to develop a predictive model for runtime based on network counters. Our model performs well, with a mean squared prediction error of 0.22.